Method, apparatus, and manufacture for a tracking camera or detector with fast asynchronous triggering

ABSTRACT

An image projection device for displaying an image onto a remote surface. The image projection device employs a scanner to project image beams of visible light and tracer beams of light onto a remote surface to form a display of the image. The device also employs a light detector to sense at least the reflections of light from the tracer beam pulses incident on the remote surface. The device employs the sensed tracer beam light pulses to predict the trajectory of subsequent image beam light pulses and tracer beam light pulses that form a display of the image on the remote surface in a pseudo random pattern. The trajectory of the projected image beam light pulses can be predicted so that the image is displayed from a point of view that can be selected by, or automatically adjusted for, a viewer of the displayed image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation application of U.S. Non-Provisionalapplication, titled “METHOD, APPARATUS, AND MANUFACTURE FOR A TRACKINGCAMERA OR DETECTOR WITH FAST ASYNCHRONOUS TRIGGERING,” Ser. No.16/435,392, filed Jun. 7, 2019, now issued as U.S. Pat. No. 10,962,867issued Mar. 30, 2021, which is a Continuation application of U.S.Non-Provisional application, titled “METHOD, APPARATUS, AND MANUFACTUREFOR A TRACKING CAMERA OR DETECTOR WITH FAST ASYNCHRONOUS TRIGGERING,”Ser. No. 15/444,182, filed Feb. 27, 2017, now issued as U.S. Pat. No.10,331,021 issued Jun. 25, 2019, which is a Continuation application ofU.S. Non-Provisional application, titled “METHOD, APPARATUS, ANDMANUFACTURE FOR A TRACKING CAMERA OR DETECTOR WITH FAST ASYNCHRONOUSTRIGGERING,” Ser. No. 14/218,643, filed Mar. 18, 2014, now issued asU.S. Pat. No. 9,581,883 issued Feb. 28, 2017, which is a Continuationapplication of U.S. Non-Provisional application, titled “METHOD,APPARATUS, AND MANUFACTURE FOR A TRACKING CAMERA OR DETECTOR WITH FASTASYNCHRONOUS TRIGGERING,” Ser. No. 13/858,762, filed Apr. 8, 2013, nowissued as U.S. Pat. No. 8,696,141 issued Apr. 15, 2014, which is aDivisional application of U.S. Non-Provisional application, titled“PHOTONJET SCANNER PROJECTOR,” Ser. No. 13/605,948, filed Sep. 6, 2012,now issued as U.S. Pat. No. 8,430,512 issued Apr. 30, 2013, which is aContinuation application of U.S. Non-Provisional application, titled“IMAGE PROJECTOR WITH REFLECTED LIGHT TRACKING,” Ser. No. 12/249,899,filed Oct. 10, 2008, now issued as U.S. Pat. No. 8,282,222 issued Oct.9, 2012; which claims the benefit of U.S. Provisional application,titled “NOVEL PROJECTION SYSTEM USING A HIGH-SPEED PSEUDORANDOM SWEEPINGLIGHT BEAM,” Ser. No. 60/998,520, filed on Oct. 10, 2007, U.S.Provisional application, titled “PHOTONJET SCANNER-PROJECTOR,” Ser. No.61/000,238, filed on Oct. 23, 2007, U.S. Provisional application, titled“PHOTONJET TRACKING CAMERA,” Ser. No. 61/002,402, filed on Nov. 7, 2007,and U.S. Provisional application, titled “PHOTONJET SCANNER-PROJECTORSYSTEM,” Ser. No. 61/005,858, filed on Dec. 7, 2007, the benefit of theearlier filing dates of which is hereby claimed under 35 U.S.C. § 119(e)and the contents of which are each further incorporated by reference intheir entirety.

TECHNICAL FIELD

The present disclosure is directed to an image projector in general, andmore particularly, to an image projector that employs observedreflection of light beams to predict the subsequent position of lightbeams that display the image on a remote surface.

BACKGROUND

With the ubiquity of images that are available for display by anelectronic device, the capabilities of a particular electronic device'sdisplay has become a significant factor to users. These images caninclude, movies, videos, podcasts, television, pictures, cartoons,illustrations, graphics, tables, charts, presentations, and the like.Also, the quality, resolution, and type of display for images that canbe displayed by an electronic device is often the primary factor in auser's decision to purchase that particular electronic device. Forexample, users' might prefer relatively low power projection displaysfor mobile devices, such as mobile telephones, notebook computers, handheld video game consoles, hand held movie players, personal digitalassistants (PDA), and the like. These low power projection displays caninclude, and backlit or non-backlit Liquid Crystal Displays (LCD).Further, other relatively low power emissive displays such as OrganicLight Emitting Diodes (OLED), are growing in popularity for mobiledevices. Also, the size of a display for a mobile device is oftenlimited to a relatively small area, i.e., displays that can easily fitin a hand or clothing pocket. The relatively small size of displays formany mobile devices can also limit their usability for someapplications.

Stationary electronic devices, such as personal computers, televisions,monitors, and video game consoles, often employ high power projectiondisplay technologies, such as Gas Plasma, Cathode Ray Tubes (CRT), LCD,DLPs (Digital Light Processor), and the like. Also, displays for theserelatively stationary electronic devices are often considerably largerthan those displays employed with mobile devices, e.g., projectiondisplays can be five feet across or more. However, the relatively largephysical size of the cabinetry associated with most displays employedwith stationary devices can be inconvenient and unattractive for manyusers, especially when the displays are not in use.

Front image projection devices can also be used to display images on aremote surface, e.g., a hanging screen of reflective fabric, or someother relatively vertical and reflective surface such as a wall. Also, avariety of different technologies are employed by front image projectiondevices, such as Digital Light Processors (DLP), Light Emitting Diodes(LED), Cathode Ray Tubes (CRT), Liquid Crystal Displays (LCD), LiquidCrystal on Silicon (LCoS), MicroElectroMechanicalSystems (MEMS)scanners, and the like. However, artifacts in the display of imagesprojected on remote surfaces have been difficult to compensate for, andoften adversely effect the quality, resolution, and usability of theseremotely projected images.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present invention aredescribed in reference to the following drawings. In the drawings, likereference numerals refer to like parts through all the various figuresunless otherwise explicit.

For a better understanding of the present disclosure, a reference willbe made to the following detailed description, which is to be read inassociation with the accompanying drawings, wherein:

FIG. 1 is a block diagram of one embodiment of a Image Projector Device(IPD);

FIG. 2 shows one embodiment of a client device that may be included in asystem implementing aspects of the invention;

FIG. 3 shows one embodiment of a IPD control sub-system;

FIG. 4A shows one embodiment of light beam trajectories generated by aIPD;

FIG. 4B shows an embodiment of one light beam trajectory with scannedand predicted trajectory portions;

FIG. 5 shows an embodiment of a IPD depicting one image beam and tracerbeam point;

FIG. 6A shows an embodiment of an application of an IPD with differentuser vantage points;

FIG. 6B shows an embodiment of an IPD response to different viewingperspectives;

FIG. 6C shows an embodiment of a IPD projection onto a tilted screen;

FIG. 7A shows an embodiment of a mobile device with an embedded IPD;

FIG. 7B shows another embodiment of a mobile device with an embedded IPDand a head-mounted position sensor; and

FIG. 8A shows a flow diagram of one embodiment of a high level processof generating an image using a IPD; and

FIG. 8B shows a flow diagram of one embodiment of a detailed process ofgenerating an image with a IPD.

DESCRIPTION OF THE EMBODIMENTS

The present disclosure will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific exemplary embodiments bywhich the invention may be practiced. This invention may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the invention to those skilled in the art.Among other things, the present invention may be embodied as methods ordevices. Accordingly, the present invention may take the form of anentirely hardware embodiment, an entirely software embodiment or anembodiment combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may. As usedherein, the term “or” is an inclusive “or” operator, and is equivalentto the term “and/or,” unless the context clearly dictates otherwise. Theterm “based on” is not exclusive and allows for being based onadditional factors not described, unless the context clearly dictatesotherwise. In addition, throughout the specification, the meaning of“a,” “an,” and “the” include plural references. The meaning of “in”includes “in” and “on.”

The term “pseudorandom” as used herein, indicates a statistically randomprocess, which appears to be random as distinguished from a truly randomprocess, which lacks predictability. Pseudorandom processes may begenerated using some deterministic elements combined with somenon-repeating patterns. For example, a timestamp may be combined withthe changing contents of a predetermined memory location to generate apseudorandom number. Furthermore, this term as used herein, indicateslack of a predefined scanning organization, for example, an array ofpixels (picture elements), parallel scanlines, or any other arrangementhaving a temporal or spatial relationship between scanlines.

The term “corresponding” as used herein, may indicate a one-to-one,one-to-many, or many-to-one mapping. For example, an image pixelcorresponding to a display position may indicate that the image pixelcorresponds to more than one display position, when adjacent displaypositions are resolvable at a higher resolution in comparison with theresolution of the image associated with the image pixel. In such a case,a single image pixel may be interpolated or resolved to cover more thanone display position, and thus, the single image pixel corresponds tomore that one display position and indicates a one-to-many mapping. Inanother case where the image pixels and the display positions havesubstantially equal resolutions, the correspondence there between can bea one-to-one mapping. Furthermore, if a plurality of adjacent pixelshave substantially equal values, they may be projected as a line to formthe image instead of a series of individual positions on a display.

The following briefly describes the embodiments of the invention inorder to provide a basic understanding of some aspects of the invention.This brief description is not intended as an extensive overview. It isnot intended to identify key or critical elements, or to delineate orotherwise narrow the scope. Its purpose is merely to present someconcepts in a simplified form as a prelude to the more detaileddescription that is presented later.

Briefly described, the invention is directed to an image projectiondevice for displaying an image onto a remote surface. The imageprojection device employs a scanner to project image beams of visiblelight and tracer beams of light onto a remote surface to form a displayof the image. The device also employs a light detector to sense at leastthe reflections of light from the tracer beams incident on the remotesurface. The device employs the sensed tracer beam light to predict thetrajectory of the beam and subsequent image beams that form a display ofthe image on the remote surface. Additionally, in at least oneembodiment, the light detector can also sense the reflections of visiblelight that form the projected image incident on the remote surface. Andthe device can employ the sensed reflections of this visible light toeffect adjustments to subsequent image beams that form the display ofthe image on the remote surface.

The trajectory of the projected image beams can be observed so that theimage is displayed from a point of view that can be selected by, orautomatically adjusted for, a viewer of the displayed image. Also, theimage beams and tracer beams that form the projected image can beprojected onto the remote surface in a pseudo random pattern that isprimarily based on the projected image itself rather than a predefinedpattern, such as an array or grid.

To display the projected image on a remote surface, the image projectiondevice projects pulses of visible light onto the remote surface and alsoprojects other pulses of tracer beam light onto the same remote surface.These pulses can be generated by modulating the output of light sourcesor shuttering the detector of the reflected light from the remotesurface. The light from the image beam and the tracer beam can becombined and projected onto the same location at the remote surface, orthey may be projected onto two adjacent locations where the distancebetween the two locations is predetermined. The image beam lighttypically has a wavelength in the visible spectrum. Also, the tracerbeam light can have a wavelength in either the visible wavelength oranother wavelength in the visible light spectrum. If visible light isemployed for the tracer beam pulses, the frequency of the pulses istypically so fast that the tracer beam pulses are undetectable to thetypical viewer. Also, if non-visible light is employed for the tracerbeam pulses, the light wavelength can be in one or more non-visiblespectrums, e.g., infrared, ultraviolet, or the like.

Furthermore, in at least one embodiment, the image projection deviceenables one or more safety interlocks between the image projectiondevice and the remote surface. For example, a determination is made asto whether or not the scanner is properly moving, and if not, the imagebeam pulses and tracer beam pulses are disabled until the scanner isproperly moving again. Also, in at least one embodiment, if a movingobject is detected between the image projection device and theprojection of the image on the remote surface, the image beam pulses andtracer beam pulses are disabled until the moving object is no longerpresent.

In at least one embodiment, the tracer beam non-visible light pulses areIR (Infra-Red) light that is projected by a scanner, such as anoscillating MEMS scanner, onto particular positions in a pseudorandomscanline trajectory that sweeps across the remote surface. A lightdetector, such as a camera, coupled with a processing unit, tracks thetracer beam light pulses to obtain data for detecting N consecutivepreceding positions of the pseudorandom scanline trajectory andestimating the subsequent consecutive M positions of the scanlinetrajectory with respect to a current screen position. Multiple componentimage beams of visible light, for example, RGB (Red-Green-Blue) colors,are modulated, based on a corresponding M pixels of an image in a memory(memory image), and combined to create discrete combined image beampulses. The combined image beam pulses are projected onto the remotesurface at the subsequent estimated positions after the current combinedimage beam position. Furthermore, the reflections of the projectedcombined image beams incident on the remote surface, in addition to thetracer beam, may be optionally detected and compared with a copy of theimage in a memory. And based on that comparison, the subsequentestimated positions of the projected image beams can be adjusted onsubsequent pseudorandom sweeps, progressively, to improve the projectedimage's sharpness, brightness, and color. The combined image beam mayalso include information about a known image to be displayed.

For hundreds of millions of users using billions of computing devices,the availability of a physically small, low-power, high-quality, highreliability, high resolution projection display with few or no visualartifacts may provide a significant advantage. These advantages may beparticularly significant for devices with small displays, such as mobilephones and PDA's. With a small, low-power, built-in micro projector,such small computing devices can display information, effectively, on alarge, high-resolution screen without the large physical size and highpower consumption rate normally associated with a large screen.

Power efficiency is also very important, particularly for small personaldevices, because of the limited battery life with daily use. Efficientprojection of light with limited use of filters, polarizers, and thelike, reduces overall power consumption, while providing a high qualitydisplay. The mixing of light falls into at least two general categories:additive mixing and subtractive mixing. In additive mixing, componentlight signals are combined with each other. In subtractive mixing, somelight frequency components are filtered out, transmissively orreflectively, or subtracted from the original whole. Some of theconventional display technologies, such as LCD and DLP, use subtractivemixing as a basic part of their operation. Subtractive mixing of lightbeams is generally wasteful of energy because light (photons) is firstgenerated and then partially blocked (subtracted), wasting the alreadygenerated light. Subtractive mixing is used to increase image brightnessand enhance contrast between the brightest and darkest pixels. Indisplay systems that use subtractive mixing, often just five percent(5%) of the energy used to generate the light is eventually utilized fordisplaying the image, resulting in poor overall efficiency.

Another important aspect of a display technology is reliability. Withfew moving parts, low power consumption, and low heat generation,reliability of invention may be generally greater compared to otherdisplay technologies having similar quality.

The feedback aspects, for both tracer pulses and image beam, of theinvention enables uses in applications that are not possible, or aremore difficult to realize, with other technologies. For example, theinvention allows automatic adjustment of a display in real-time based ona perspective/position of the viewer/user. The user may move around aremote display screen while the invention automatically adjusts thedisplayed image to provide the appropriate perspective as viewed fromeach new position of the user with respect to the direction ofprojection onto the screen. This feature may have uses in immersiveapplications, such as video games. A variation of this capability isthat the projected image may be displayed on an un-even surface of ascreen, such as a textured wall, fabric, or other background withtexture or curvatures that would otherwise distort the projected image.These features are more fully described below with respect to FIGS. 3-7.

Even though various embodiments refer to the RGB color space, othercolor spaces may also be used. For example, YIQ, YUV, YCbCr, and thelike, color spaces may also be used to provide an image for projection.Similarly, more than three basic colors may be used. For example, inaddition to the Red Green Blue color sources, other emissive spectralcomponent color sources may be used, such as an Orange color source, forexample, in the wavelength range of 597 to 622 nano-meters (nm), aYellow color source, for example, in the wavelength range of 577 to 597nm, or a Violet color source, for example, in the wavelength range of380 to 455 nm. In this way, four or more component color sources may beused to project the image on a remote surface.

Generally, use of additional component colors may remove some tradeoffs,for example, the tradeoffs between efficiency, efficacy (characterizedas perceived lumens per watt), gamut (broadest color range), andfidelity, characterized by avoidance of perceptive artifacts, such asspatial, temporal, motion, or color artifacts. Such tradeoffs may haveto be made in both spatially rasterized and field-sequential colordisplay systems, like LCDs, having addressable display elements orpixels. The use of a greater number of coherent monochrome sources, suchas laser beams, reduce speckle that may be caused by self-interferencein the reflected beam of each component light beam. The greater numberof component light beams may reduce speckle because the component lightbeams are not coherent with respect to each other and may cancel out.

Multiple component light sources of substantially identical wavelengthmay also be used for added peak power and efficiency. For example,multiple semiconductor light sources, such as LEDs and laser LEDs, maybe optically combined to generate a brighter and more efficient lightbeam. Speckle may be reduced due to reduced phase coherency in such“gang source” systems where multiple near-identical wavelength componentlight sources are used.

Illustrative Operating Environment

Electronic displays for computing devices, such as workstations, desktopand laptop Personal Computers (PC), mobile devices like mobile phones,PDA's, and the like, as well as displays for entertainment-orienteddevices, such as televisions (TV), DVD players, and the like, may bereplaced or supplemented by image projection device (IPD). In oneembodiment, the IPD is an integral part of the computing orentertainment device. In another embodiment, the IPD may be asupplemental external display device used in addition to, or in placeof, a conventional display.

One embodiment of a computing device usable with the IPD is described inmore detail below in conjunction with FIG. 2. Briefly, however, thecomputing device may virtually be any stationary or mobile computingdevice capable of processing and displaying data and information. Suchdevices include mobile devices such as, cellular/mobile telephones,smart phones, display pagers, radio frequency (RF) devices, infrared(IR) devices, PDAs, handheld computers, laptop computers, wearablecomputers, tablet computers, mobile video game consoles, integrateddevices combining one or more of the preceding devices, or the like.Stationary computing devices may also include virtually any computingdevice that typically connects using a wired or wired communicationsmedium such as personal computers, video game consoles, multiprocessorsystems, microprocessor-based or programmable consumer electronics,network PCs, or the like.

Computing devices typically range widely in terms of capabilities andfeatures. For example, a mobile phone may have a numeric keypad and afew lines of an LCD or OLED display on which a small amount of text andgraphics may be displayed. In another example, a computing device mayhave a touch sensitive screen, a stylus, and a relatively large displayin which both text and graphics may be displayed.

A computing device may include a browser application that is configuredto send/receive and display web pages, web-based messages, or the like.The browser application may be configured to receive and displaygraphics, text, multimedia, or the like, employing virtually any webbased language, including a wireless application protocol messages(WAP), or the like. In one embodiment, the browser application isenabled to employ Handheld Device Markup Language (HDML), WirelessMarkup Language (WML), WMLScript, JavaScript, Standard GeneralizedMarkup Language (SMGL), HyperText Markup Language (HTML), eXtensibleMarkup Language (XML), or the like, to display and send information.

Communication media used with computing devices typically may enabletransmission of computer-readable instructions, data structures, programmodules, or other types of content, virtually without limit. By way ofexample, communication media includes wired media such as twisted pair,coaxial cable, fiber optics, wave guides, and other wired media andwireless media such as acoustic, RF, infrared, and other wireless media.

The display of images and/or video data using the IPD is different fromthe display of the same images/video on more conventional displays.Conventional displays generally are arranged with some form of pixeladdressing in the display area. The pixel address may be specified bytiming, as in raster scanners like CRTs using horizontal and verticalblanking signals (H-sync and V-sync, respectively), or by row-columnaddress pairs like transistor/LED (Light Emitting Diode) arrays in LCDs.In conventional displays, the area of the display is generally quantizedas a fixed grid with equal sized tiles or spatial quanta, also referredto as pixels. In such display systems, the illusion of motion is createdby quantizing continuous time into discrete and equal quanta, alsoreferred to as frames. Generally, a fixed frame rate, expressed asFrames Per Second (FPS) is used to record and play back moving images.This quantization of time into frames, and image into pixels, introducetemporal and spatial artificial visual artifacts, respectively, duringthe display of moving images, such as jagged lines (spatial), aliasing(spatial and temporal), image blurring (temporal), judder (temporal),and the like, further described below.

These address-based pixel organizations are fundamentally different fromthe pseudo random scanning method used in IPD. The address-baseddisplays generally require a defined frame format with specific andrigid timing requirements, as is well known to one skilled in therelevant arts. For example, in a raster scanner, scanned lines(scanlines) are displayed in consecutive order, as parallel horizontallines, one after another from the top to the bottom of the screen. Incontrast, the MEMS scanner for the IPD can oscillate and project lightin a pseudorandom pattern onto a remote surface, where the scanlines ofimage beam and tracer beam light pulses are primarily traced out in adirection based on the image to be projected, without relying upon aparticular spatial or temporal relationship with the previous or thenext scanline.

Conventional scan patterns, in terms of both timing and spatialregularity, digital sampling, and quantization in general create anumber of visual artifacts, such as jagged slant or diagonal lines,especially visible in low-resolution displays, image blurring, motionblur (may occur because of a basic mismatch between continuous humanvision and quantized digital display), judder (defined as smallunnatural jerky movements in motion pictures, either in space or intime. In space, judder can be the result of consecutive film frames notadvanced precisely to the same position at the projector gate. In time,judder in video may be noticed because 24 frames per second for filmsource does not divide evenly into 60 fields or frames per second forNTSC video, and some film frames' content is shown on the screen formore time than other frames' content), moirés (pattern resulting fromtwo grids that are superimposed over one another), screen door effects,aliasing, and the like. These artifacts results, in one way or another,from quantization and digital sampling.

However, since the IPD is relatively analog in nature and in itsoperation as compared to conventional displays, does not depend onquantization, even though at some points through the process a IPDimplementation can process digital data. For example, reading an imagefrom memory to display may involve some digital processing, however,once the image is ready for projection using IPD, the process is largelyanalog, as more fully described herein.

Another difference between the IPD and conventional displays is trackingand feedback. Conventional displays are feed-forward in their basicoperation, sending image data in one direction: from the source of theimage, for example, memory or DVD, to the destination, which is thedisplay device. Generally, no feedback is needed for basic operation andno data is read back from the conventional display. In someimplementations of conventional display devices, the displayed image maybe read back or sensed and compared with the source image to increasethe quality of the displayed image. However, such feedback is forquality enhancement as opposed to being part of the basic operation ofthe conventional display devices. The display method used in IPD may usefeedback from the tracer beam, for example, IR pulses, to determine thenext screen position on the scanline trajectory by trajectory predictionand/or estimation. Once the next screen position is so predicted, thememory image for the corresponding screen position is obtained and usedto modulate the component image beams for combining and projecting ontothe next screen position.

Because the scanlines can be pseudorandom in the IPD, generally, thetiming information (usually included in video frames) needed for displayof a video stream on a conventional display device, may not be neededfor display of the same video stream on the IPD. In one embodiment, theactive video signal, the sequence of images to be displayed, that issent to a conventional display screen, may be stripped, in real-time,from the formatting information associated with the conventionaldisplays, such as H-sync and V-sync signals and/or other timinginformation, for display using IPD. In another embodiment, the activevideo or other image to be displayed using IPD may be generated for IPDor pre-stripped at the source of the video or image. In yet anotherembodiment, both of the above embodiments may be implemented bydynamically determining whether an image has display format informationor is pre-stripped. In still another embodiment, the timing informationincluded in video frames may be used to extract information that may beused to improve quality and/or perceived resolution. For example, aweighted average of multiple frames may be used to insert preciseinterpolated inter-frame pixel values that reduce unwanted visualartifacts.

Many graphics programs, such as video games and programs for 3-D (threeDimensional) manipulation of objects, may generated images witheffective resolutions that are significantly beyond the resolutions thatcan be displayed by conventional display devices. Such high resolutiongraphics generated by these graphics programs may specially benefit froma projection system, such as the PSTP, that is substantially free fromquantization effects and has very high effective resolution.

FIG. 1 is a block diagram of one embodiment of IPD 100. This embodimentincludes a light source driver 102 for modulating component image beams,generated by light sources 104, and for controlling tracer beamgenerator 106. The component image beams are combined using a beamcombiner 108 that produces a combined image beam 120. The combined imagebeam is directed to a scanner 110, which reflects the combined imagebeam to a screen 114. Combined image beam 120 includes information abouta known image to be displayed. The information about the known image isprojected sequentially (serially) onto screen 114 using the consecutiveprojected scanlines causing the known image to appear on screen 114. Theformation of the known image on the screen may take a few microsecondsor less.

Tracer beam 122 used for scanline trajectory prediction is also directedto scanner 110 and reflected to screen 114. Detector 112 is used todetect the reflection of the tracer beam off screen 114. Detector 112sends timing to and screen position information [x, y] to processor 116coupled with memory 118 holding an image to be displayed on screen 114.Detector 112 can also be used to optionally detect the reflection of theimage beam off screen 114. Additionally, in at least one embodiment, aseparate detector 113 can be included to separately detect thereflection of the image beam off screen 114. Process 116 is also couplewith light source driver 102 to control the modulation of componentimage beams, and generation of tracer beam 122 by tracer beam generator106.

In one embodiment, component light sources 104 include RGB. In anotherembodiment, component light sources 104 may include other colorcomponents such as orange, yellow and violet in addition to RGB. In oneembodiment light sources 104 are LEDs (Light Emitting Diode), while inanother embodiment, light sources 104 are lasers. In yet anotherembodiment, light sources 104 are laser diodes. Those skilled in therelevant arts will appreciate that many types of light sources may beused to generate component lights, such as red, green, and blue, and thelike, without departing from the spirit of the disclosure.

Component light sources 104 produce component image light beams that arecombined by a beam combiner 108 to produce a combined image beam. In oneembodiment the beam combiner 108 is an optical device, such as a prism,dichroic mirrors, or the like. In another embodiment, the beam combiner108 is an electronic device that may be used to convert light componentsinto electrical signals, mix the electrical signals into a mixed signal,and convert the mixed signal back into a combined image beam. Anelectronic mixer may be used if intermediate processing of the lightbeam, such as digital or analog filtering or other control andprocessing, is desired.

In one embodiment, scanner 110 may be a MEMS device with a precisionbuilt mirror with at least a two-axis gimbal for independentlycontrolled rotation about two orthogonal axis. In this embodiment, themirror may pseudorandomly project scanlines covering any screen positionon the surface of screen 114. In another embodiment, scanner 110 may bea mirror with other types of directional controls, such that a scanlinemay be projected on every screen position on screen 114. For example,polar or cylindrical adjustments and corresponding controls may be usedto point the mirror to direct the reflection of image beam to any screenposition on screen 114. Because of the rotation of the mirror, ascanline projected on screen 114 may have a slight curvature instead ofbeing a straight line. Scanner 110 may work in a color sequential mode,where the image beam sequentially varies across multiple colors and theimage is reconstructed by the viewer by virtue of time integration ofthe various color values observed.

In another embodiment, that may be useful for highly mobile images, suchas movies, a multiple-color system may be implemented using multipleprimary colors simultaneously. The primary colors may be separated outby using a prism, dichroic beam splitters or dichroic mirrors, or byusing separate sub-pixels with color filters. In another embodiment, afull, broad-spectrum white source may be used for illumination, and thewhite beam may then be split into its color components which areseparately observed by multiple lines in a linear array sensor.

A particular scanline projected by the scanner 110 onto the screen 114is not aimed, in a feed-forward fashion, at any particular predeterminedposition (or curve, when referring to all points on the scanline) on thescreen 114. Rather, the scanline is pseudorandomly projected at somearbitrary position on the screen 114 and the arbitrary position isobserved and detected by a detector, more fully described below. Thisfeedback arrangement is generally more precise and accurate thanfeed-forward, because the actual position of the projected scanline onthe screen is determined, instead of a predicted feed-forward position,which may be off due to many causes, such as vibration of the scanner110 and/or the screen 114. Feedback inherently makes image correction onthe screen possible, for example, to counter screen imperfections and/orvibrations, because feedback is after-the-fact observation of an event(e.g., scanline position on the screen), rather than before-the-factprediction and/or specification of the event, as is the case infeed-forward systems.

In one embodiment, screen 114 is a typical front projection screen witha high reflexive index. In another embodiment, screen 114 is a backprojection screen with diffuse transmittance for passing light through.In this embodiment, the other components shown in FIG. 1 are arranged toproject the image onto the back of the screen to be viewed from front ofthe screen by a viewer. In another embodiment, screen 114 may be a lightcolored wall or any other flat surface. In yet another embodiment,screen 114 may be any surface with or without texture. The feedbackfeature of PSTP 100 may automatically compensate for surfaceimperfections, texture, and irregularities of screen 114.

In one embodiment, detector 112 is a monochrome camera. The monochromecamera detects single colors, such as red, green, or blue. Monochromecameras may also detect IR light. A monochrome camera may be useful whenthe tracer beam 122 is a visible pulsed light beam. In this embodiment,the light beam pulses are of short enough duration that would beimperceptible to the human eye. In another embodiment, detector 112 maybe an IR detector used to detect pulses projected by the tracer beam122. In yet another embodiment, detector 112 may be a CCD (ChargeCoupled Device) array. The CCD array may be a single row CCD array or itmay be a two dimensional CCD array.

In yet another embodiment, multiple single row CCD arrays may be used.In this embodiment, a two-dimensional projected image is opticallycollapsed into two orthogonally related linear image components. Each ofthe two linear image components is detected by separate single row CCDarray. This way, a target object on the projected image may be detectedand tracked. In the past, a camera system constructed from speciallyadapted optics and sensors—could be optimized to detect and track one ormore target objects, for example, dots or small bright features, in abroad field of view. With simple mirror optics a single linear arraysensor is sufficient to detect both location and signal intensity of thedots in a 2D field with a minimum of computation. Positionalobservations can be made at very high speed with an accuracy notattainable with conventional camera systems. However, the IPD eliminatesthe trade-off between spatial resolution and shutter speed as observedin existing array sensors.

In still another embodiments, if a singular target object, for example,a single screen position the light level of which exceeds a pixelthreshold of a detector array (for example, CCD array), a folding opticsarrangements may be used to superimpose a dimension Y (for example,vertical) of an image onto a dimension X (for example, horizontal) ofthe image so that information contained in both dimensions may bemeasured and detected by a single linear CCD array. U.S. ProvisionalPatent application, Ser. No. 61/002,402, referenced above and to whichpriority is claimed and incorporated by reference, discloses thisarrangement. The folding optics arrangement may be implemented using amirror, a half-mirror (for beam splitting), or a reflecting prism, amongother options. In one embodiment, pixels in the detector 112 array mayhave the same associated threshold. In another embodiment, pixels in thedetector 112 may have a different associated threshold based on certaincriteria, for example, the contrast expected in the image that will bedisplayed. In yet another embodiment, the pixel thresholds may be setdynamically based on certain criteria, for example, the average darknessor brightness of the image about to be displayed. The dynamic setting ofthe pixel thresholds may be performed during a calibration cycle ofdetector 112, where the pixel threshold value is set to just below thebrightest value observed in the detector sensor array.

The folding optics embodiment of detector 112 enables out-of-orderreporting of screen positions exceeding the pixel threshold.Additionally, this embodiment enables dimensional superposition, as morefully described below. In one embodiment, each pixel in detector 112array may be set to a threshold. The threshold is set above the naturalnoise level, for example, the dark current of the electronicsimplementing the detector pixels. The threshold may also be set at ahigher level than the natural noise level, but below the expectedbrightness of the target object to be detected. This way, the brightestscreen position in the projected image exceeds the pixel threshold andtriggers detection of the screen position. Each pixel in the detector112 array has a predetermined address (pixel address) typicallycorresponding to a sequential count that can either be hard-wired orassigned during initialization of the projector system.

When a pixel in detector 112 array exceeds the threshold value, a pixel“hit” event, the address of the pixel is reported, for example toprocessor 116 (see FIG. 1), together with the measured light value forthe corresponding screen position. For example, the data that isreported to processor 116 may take the form of a 2-tuple [Address ofpixel N, Value of pixel N]. Either concurrently or afterwards, the pixelis reset, so the pixel is ready to detect the next event, for example,another screen position the brightness of which exceeds the pixelthreshold. All other pixels in detector 112 array are unaffected duringthe detection reset cycle and pixels in detector 112 array may operatefully asynchronously. Multiple pixel hit events may be detected at thesame time.

In this embodiment, no electronic or mechanical shutter may be neededfor detecting pixel hit events. The pixel thresholds and resettingtogether act as an effective shutter mechanism for detector 112 array.An effective shutter speed of a sensor (pixel) in detector 112 array isat most as slow as the reset time of any one pixel, typically on theorder of a few micro seconds or less. The shutter speed for thedetection of a pixel hit event, may generally be limited by thesensitivity of a pixel trigger circuit to detect a minimum amount ofcharge exceeding the preset threshold value (trigger). If the value tobe detected is a sufficiently bright strobe, the detection would benearly instantaneous. For example, an avalanche photo diode detectorbased pixel might require as few as 30 photons to detect a hit. Whenhigh speed beam detection is required, adjacent pixels in the detectorcorresponding to screen positions on the scanline, may be triggered,effectively causing the detector to have little or no shutter speedlimit. Furthermore, nearly simultaneous observations of different pixelhit events may be pipelined for processing to calculate and predict thetrajectory of the scanline. A precise reference clock may be used totime-stamp each pixel hit event observation for accurate trajectorycalculations. Such a reference clock may provide GHz (Giga Hertz;nano-second time resolution) time-stamps.

The screen position, brightness, and timestamp associated with a pixelhit even may be obtained asynchronously. Therefore, the timing of theobserved pixel hit events is independent of sampling speed, samplingintervals, or shutter speed of detector 112, thus, avoiding measurementerrors and quantization effects, which typically result in imageartifacts such as aliasing in traditional array sensors. Reducingquantization artifacts is an important architectural consideration. Inthe cases that the observed object has a known periodicity, clockaccuracy and motion accuracy can be derived independently from thesensor spatial resolution, reducing motion blur. In one embodiment, thetracer beam 122 (see FIG. 1) may be pulsed with a low duty cycle, on theorder of pico-seconds. Such low duty cycle IR pulses create ashort-duration bright flash and ensure that the screen position at whichthis bright flash appears do not spread across more than any twoadjacent pixels in detector 112 array, even when the scanline sweepvelocity may far exceed the spatial accuracy and shutter speed of aconventional camera.

In one embodiment, dimensional Superposition entails beam folding inaddition to “deconstructing” the image, e.g., optically reducing theimage dimensionally from a 2-D space, having an X- and a Y-dimension,into two or more 1-D spaces. In this embodiment an N×N image detectorarray may be reduced to a single N-pixel linear sensor array by“folding” and superimposing each of the two dimensions of the image ontoone linear array. A 2-D image beam may be collapsed into two 1-D lineararrays using one or more cylindrical lenses that split the 2-D imagebeam. Beam folding may be performed by guiding the image light exitingfrom a cylindrical lenses onto a single linear array by optical means,for example, using mirror(s), prism(s), grating(s), or waveguide(s). Inone embodiment, the single linear array may be a monochrome lineararray. In another embodiment, the 2-D image may be split, resulting intwo split image beams. One of the split beams may be rotated andrecombined with the other split image beam, so that one cylindrical lensand corresponding sensor array may be needed.

In this embodiment, a single screen position where the IR pulse isprojected on the screen, may result in two separate illuminated dots onthe linear array. One dot represents X-dimension and the other dotrepresents the Y-dimension of the screen position on the original 2-Dimage. Thus, the bit values (representing, for example, brightness) andaddresses of the two dots in the single linear array allow the spatialscreen position in the 2-D image, as well as the intensity and color ofthe 2-D image corresponding to the screen position be reconstructionrapidly and efficiently, as two values have to be read from the samesensor in the single linear array. This embodiment allows the projectionsystem to track a highly mobile bright object on the screen with N×Nspatial resolution at a speed limited by the pulse width (or period) ofthe tracer beam. In this case, the effective resolution of the PSTP isequivalent to speed x pulse period. In one embodiment, for accuratescanline trajectory calculations, multiple sequential readings can bemade as described and the result can be hyper-resolved by interpolationfor scanline sweep segments with known speed and smooth curvature. Thisway, the PSTP scanner may achieve high resolution and great color depth,using a single monochrome array sensor.

In one embodiment, improved spatial color accuracy, for example, hue andluminance, may be independently achieved, using the same set of opticsand sensors. To detect the screen position accurately based on areflection of incident beams projected on the screen 114, a narrow bandfrequency light source may be used, in the form of a narrowly collimatedillumination beam that illuminates a very narrow spot, for example, anIR or UV beam. If all light beams have substantially the same frequency,the optics may be optimized or tuned for this frequency. A broadspectrum light source or a mix of multiple primary color components,representing the image to be displayed, may be superimposed andprojected onto the screen position to determine the hue and luminanceinformation of the reflected image beam. To enhance the sensitivity ofthe color measurements, and to allow for some blurring and coloraberrations inherent in using the broad spectrum light in the optics ofthe detector 112, values of multiple pixels in the single linear arraymay be combined. This may be possible because due to the lasercollimation accuracy, the image information may be generally centered onthe same location as the narrow beam location, which has been alreadydetermined.

In one embodiment, both the screen position an and visible image colorcomponents may be read simultaneously by subtracting IR pulse value ofthe screen position from sum of the adjoining “lit-up” screen positions.In another embodiment, the periodic IR pulse may be used to determinethe precise scanline trajectory. The scanner may operate in colorspatial field sequential fashion, where for each color, location iscalculated by spatial interpolation based on the time-stamp associatedwith the screen position.

In one embodiment, processor 116 is a programmed microprocessor coupledto memory 118 containing the image to be displayed. As is well known tothose skilled in the art, embedded programs running on themicroprocessor may be used to perform appropriate signal processingusing processor 116. In one embodiment, processor 116 is a digitalsignal processor (DSP) with specialized instructions for processingsignal information. In another embodiment, processor 116 is a circuitfor performing hardware-based processing and calculations. In yetanother embodiment, processor 116 is a software component running on amicroprocessor or microcontroller running other applications as well.Those skilled in the art would appreciate that processor 116 may beimplemented using multiple processors, each processor performing aportion of the calculations needed.

In one embodiment, light source driver 102 is a circuit for modulatinglight sources 104 according to image control signals 124 output from theprocessor 116. Light source driver 102 may also process timinginformation is 126 to control tracer beam generator 106. In oneembodiment, light source driver 102 modulates light sources 104 andcontrols tracer beam generator 106. In another embodiment, light sourcedriver 102 may be implemented as two separate modules, one forcontrolling light sources 14 and one for controlling tracer beamgenerator 106.

Not all the components shown in FIG. 1 may be required to implement PSTP100 and variations in the arrangement and type of the components may bemade without departing from the spirit or scope of the disclosure. Forexample, light source driver 102 and processor 116 may be integratedinto one device. Conversely, light source driver 102 for modulatingcomponent light sources 104 may be a device that is distinct fromprocessor 116. Similarly, in various embodiments, detector 112 andscanner 110 may be integrated together or may be implemented as discretecomponents.

In operation, with continued reference to FIG. 1, scanner 110pseudo-randomly sweeps screen 114 with scanlines. In one embodiment, arandom number generator is used to generate at least a component used toperform pseudorandom sweeps for scanner 110. In another embodiment,physical features may be used in the structure of scanner 110 tointroduce pseudo-randomness in the sweep direction of the mirroremployed in scanner 110. For example, irregular mechanical features,optical features, electrical currents, magnetic fields, or thermalfluctuations may be used to introduce such randomness. As scanner 110sweeps across screen 114, the tracer beam 122 is reflected by scanner110 and projected onto screen 114. Subsequently, tracer beam 122 isreflected back by screen 114 and is detected by detector 112. In oneembodiment, tracer beam 122 is a pulsed IR beam. In another embodiment,tracer beam 122 is composed of short duration visible light pulses.Those skilled in the art will appreciate that a behavior pattern thatmay appear random at one level of detail may be deterministic at a moredetailed level. For example, a pseudorandom behavior pattern atmicro-second level of detail may be entirely deterministic at anano-second level of detail. For instance, regular events at nano-secondtime resolution, when combined into longer units of time, may give riseto higher level events that appear random.

Detector 112 determines the screen position as defined by a pulse fromtracer beam 122. Processor 116 predicts the subsequent screen positionsbased on the previous screen positions on the same scanline. Thesubsequent screen positions are used to obtain image pixel informationfrom memory 118 and display on the subsequent predicted screenpositions. In one embodiment, the image pixel information from memory oranother image source, such as a graphic engine, may correspond to one ormore screen positions. The image to be displayed may be a described as apolygonal element or continuously interpolated color mapping. Thisprocess may be continuously repeated for pulse after pulse.

For each detected pulse, the corresponding screen position informationis provided to processor 116 by detector 112. In one embodiment, thescreen position information may include raw data as collected bydetector 112, for example, an array index of detector 112. In thisembodiment, the raw data may be further processed by processor 116 tocalculate the [X, Y] coordinates of the detected pulse. In anotherembodiment, the screen position information may include [X, Y]coordinates of the detected pulse, as determined by detector 112.Detector 112 may also provide time of detection, to, to processor 116.The time of transmission of the tracer beam from the tracer beamgenerator 106 to processor 116, the pulse “flight time,” may becalculated by subtracting the time of generation of the pulse from thetime of detection of the pulse. The pulse flight time may be used toestimate the distance of the particular point on screen 114 where thepulse was detected to scanner 110. Knowing this distance enablesprocessor 116 to adjust the brightness and modulation of the componentimage beams, and tracer beam pulse frequency to compensate for a roughor irregular surface of screen 114.

In one embodiment, successive screen positions traversed by thepseudorandom scanlines, used to project combined image beam 120 ontoscreen 114, are looked up by high speed processor 116 in a memoryreference table and the color values of the image to be displayedcorresponding to the screen positions are rendered on the screenpositions nearly instantaneously. Hardware graphics processors can lookup such color values and any related computations at multiples of GHz(Giga Hertz, a billion cycles/second) clock speeds. Thus, pixel colorvalues, typically specified as 24, 32 or 48 bit color vectors, can becomputed billions of times per second, with a latency of about a fewnano seconds (billionths of a second). This rate of processing is morethan sufficient to render the equivalent of 30 frames of two millionpixels each second (60 million vectors per second). Thus, suchcomputational speeds are more than sufficient for IPD 100 to render HDTV(High Definition TV) quality video.

In one embodiment, the projection and detection functions describedabove may be alternated on a small time scale, for example, on the orderof a few nano-seconds. In this embodiment, in a first phase, aprojection cycle is performed projecting a portion of a scanline on thescreen, while a detection cycle stands by. In a second phase, thedetection cycle starts while the projection cycle stands by. The firstand second phases may be performed on a single or a few screenpositions. In another embodiment, the projection and detection functionsmay operate simultaneously. For example, two sliding windows overlappingin time may be used to simultaneously project and detect multiple screenpositions. One sliding window may be used for projection of an imageonto multiple screen positions, while the other sliding window may beused for detection of previously projected multiple screen positions.

In its basic operation as described above, IPD 100 is independent offrames of video in contrast to conventional projectors or other displaydevices. In IPD, pixels on the projected image are refreshed not by anorderly scan, such as raster scan, but by pseudorandom scanlines. Assuch, the limitations associated with quantization/digitization, framerates, and the like, as described above, largely do not apply to IPD100. For example, HDTV needs to display about 180 million dots persecond for 30 frames per second of 1920 columns, 980 rows for eachframe, and three colors per pixel. As a result, every pixels must bere-drawn because of the regular frame rates. This type of “overkill” isnecessary to suppress some of the quantization artifacts discussedabove. In contrast, IPD 100 can update a pixel using a single combinedimage beam 120 if the pixel is predicted to fall on the currentscanline, avoiding the need to process/calculate pixels in regularsequence to get rid of artifacts.

Additionally, little or no calculations need be performed when there isa span of non-changing image value covering multiple screen positions.For example, for a solid background color the value of the image beamcorresponding to screen positions falling within the solid backgroundneed be calculated once and used tens, hundreds, or thousands of timesduring one or more scanline sweeps. Human visual system is persistent insome respects, holding a perceived brightness or color value for a smallspan of time, on the order of a few microseconds. Additionally, thehuman visual system may integrate successive values of a pixel over timeprovided the refresh rate is beyond human visual perception. The resultof such integration may be a perceived as a color different from all ofthe successively integrated values of the pixel over time. There is noconcept of frame involved in the IPD, even though a sequence of imageframes may also be displayed using the IPD like any sequence of images.

In one embodiment, the expected reflection of the image on the screen ata particular screen position may be compared to the actual reflectionvalue (for example, returned color intensity of each color of theprojection beam colors). A difference between an expected beamreflection value at each screen position and the actual reflection valueat the same screen position is an indication of thereflection/absorption response of the screen. Thus, images prior toprojection may be observed and acted upon during and after projection.

In one embodiment, the difference between the expected and the actualreflection values may be used when on-screen, location-specificreferences are required that allow the IPD to align and match itsprojection with great color and spatial accuracy to a predeterminedlocation, color, or pattern. Thus, the projected image might complement,or alternatively mask, a reflected image. For example, a screenartifact, such as a user-interface component embedded in the screen,like a microphone or a camera lens, or a non-uniform screen area may bedetected. Based on such detection, the IPD may adjust the projectedimage to avoid projecting the image on such screen artifacts. Thisfeature may be useful in a video conferencing session, where theprojected image of a first party in the conference needs to be properlyaligned with the camera aperture embedded in the screen (which may beconsidered as a small irregularity in the screen surface) so that anobserving second party to the video conferencing session looks directlyat where the first party is looking, that is, straight into his/hereyes. If this embodiment is implemented on both sides in the videoconference, the gazes and observed facial perspectives of both partiesmay be properly and mutually aligned. This feature enables a life-likeand natural eye-to-eye contact, greatly facilitating communicationbetween humans. In addition, due to the ability of the scanner to detectthe lens aperture accurately, the sweeping projection beam can bespatially masked and can be synchronized with the shutter interval ofthe camera so that the camera is not blinded by the projected image.

In another embodiment, the projected image may be aligned with theprojection surface, for example, in a situation where the projectionsurface is in motion with respect to the projector or the viewer, andthe projected image needs to be angularly and/or spatially adjusted, forexample, by rotation, translation, and/or scaling, to match the movingprojection surface. The IPD may have high scanline projection speeds,for example about 10,000 feet per second (fps). At high projectionspeeds, such image adjustments may be made faster than the human eye candetect. In this embodiment, the RSPT may create an image that seemsphysically attached to the moving projection surface. One application ofthis embodiment might be the projection of advertising on movingvehicles or objects. Another application that may require precisecontrol is an electronic dashboard display where it is desirable toalign the projected image accurately with the moving/vibrating controlsurface, particularly if that surface has an active (for example,emissive or reflective) display that needs to be complemented by theprojected image. For example, in traffic control or mission control,projecting real time flight or vehicular information on a terrain mapmay be possible using this embodiment.

In yet another embodiment, a white board or other collaborative surfacemay be used as a screen for the IPD, where written, printed, orprojected images may be annotated with real, for example, erasablemarkers. The annotations or other markings may be superimposed on aknown image that is projected onto the white board. Such markings maythen be scanned in by the IPD, so the markings can be added in real-timeto the image or document being projected onto the collaborative surface,and/or stored for later use or further modification. In one embodiment,the detector in IPD, for example detector 112 (see FIG. 1), may includea detection camera that can detect an image on the screen in addition todetecting the tracer beam pulses. In this embodiment, the differencebetween the known image projected by the IPD and the reflected imagethat is read back, or detected, by the detection camera may be used todetermine if any additional images, such as text, lines, or physicalobjects like a hand or a pointer, and the like, have been projected orotherwise marked on the screen and superimposed on the known image. Forexample, if an image of a drawing is projected on a white board and auser writes a number or text, for example, using a physical marker, onthe white board, the detection camera may detect the difference betweenthe drawing before and after the text was written by the user.Subsequently, the projected drawing may be augmented with the textwritten by the user on the same spot that the user wrote. In oneembodiment, this augmentation may be done in real-time and substantiallyimmediately. In another embodiment, the augmentation may be done laterafter storage of the detected differences.

This embodiment may be particularly useful in cases where themodifications on a real white board need to be available in real-time inmultiple locations in a collaborative setting. During a collaborationsession, an image, such as a system diagram, may be projected on aregular white board at multiple locations using the IPD. Anymodifications made to the projected diagram at any one of the locationsparticipating in the collaboration session may be seen at allparticipating locations. In another embodiment, the IPD may be a rearprojection system. In this embodiment, when using a “dry erase” markeron a translucent screen, the IPD may detect, from behind the screen, theink left by the “dry erase” marker. In another embodiment, a physicalobject, for example, a pointing device, such as a stick or a finger, maybe detected by the IPD and used to augment the projected image. Forexample, if a participant at one location uses a stick to point to afeature on the projected image, other participants at other locationswill see an augmented image including the pointer pointing to the samefeature on the image. Similarly, in an immersive video game or a virtualreality (VR) environment, physical objects, such as chairs, tables,avatars, player characters, and the like may be integrated into thescene, adding to the realism by making the VR experience more consistentwith reality.

Those skilled in the art will appreciate that the same functionalitiesdescribed above for IPD 100 may be implemented using other arrangementsof components without departing from the spirit of the presentdisclosures. For example, although IPD 100 is shown and discussed withrespect to a front projection arrangement, where the projected image isviewed by a user as reflected off screen 114, substantially the sameconcepts, components, and methods may be used for a rear projectionarrangement, where the projected image is viewed on the other side ofscreen 114 via light transmission.

Illustrative Computing Device Environment

FIG. 2 shows one embodiment of computing device 200 that may be includedin a system using IPD 100. Computing device 200 may include many more orless components than those shown in FIG. 2. However, the componentsshown are sufficient to disclose an embodiment for practicing thepresent invention.

As shown in the figure, computing device 200 includes a processing unit(CPU) 222 in communication with a mass memory 230 via a bus 224.Computing device 200 also includes a power supply 226, one or morenetwork interfaces 250, an audio interface 252 that may be configured toreceive an audio input as well as to provide an audio output, a display254, a keypad 256, a light source driver interface 258, a videointerface 259, an input/output interface 260, a detector interface 262,and a global positioning systems (GPS) receiver 264. Power supply 226provides power to computing device 200. A rechargeable ornon-rechargeable battery may be used to provide power. The power mayalso be provided by an external power source, such as an AC adapter or apowered docking cradle that supplements and/or recharges a battery. Inone embodiment, CPU 222 may be used as high performance processor 116for processing feedback and timing information of IPD 100, as describedabove with respect to FIG. 1. In another embodiment, CPU 222 may operateindependently of processor 116. In yet another embodiment, CPU 222 maywork in collaboration with processor 116, performing a portion of theprocessing used for the operation of IPD 100.

Network interface 250 includes circuitry for coupling computing device200 to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,global system for mobile communication (GSM), code division multipleaccess (CDMA), time division multiple access (TDMA), user datagramprotocol (UDP), transmission control protocol/Internet protocol(TCP/IP), SMS, general packet radio service (GPRS), WAP, ultra wide band(UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access(WiMax), SIP/RTP, Bluetooth, Wi-Fi, Zigbee, UMTS, HSDPA, WCDMA, WEDGE,or any of a variety of other wired and/or wireless communicationprotocols. Network interface 250 is sometimes known as a transceiver,transceiving device, or network interface card (NIC).

Audio interface 252 is arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 252 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action.

Display 254 may be a CRT, a liquid crystal display (LCD), gas plasma,light emitting diode (LED), or any other type of display used with acomputing device. Display 254 may also include a touch sensitive screenarranged to receive input from an object such as a stylus or a digitfrom a human hand. IPD 100 may replace display 254 or work inconjunction with display 254. For example, if display 254 is an outputor write-only device, that is, an output device that displaysinformation but does not take input from the user, then IPD 100 mayreplace display 254. However, if display 254 is an input/output deviceor read/write device, then IPD 100 may work in conjunction with display254. IPD 100 may display the output information while display 254 maytake user input, for example, as a touch-screen display. This way, auser can view high quality output using IPD 100 while inputtinginformation via the touch-screen display 254, which may additionallyoutput the same information viewed on IPD 100. In another embodiment,described more fully below with respect to FIGS. 5 and 6, the feedbackprovided by IPD 100 may be used to detect user input in 3-D, integratesuch input into the image being displayed, and project the integratedimage onto the screen 114 in real time. This embodiment reduces oreliminates the need for display 254.

Keypad 256 may comprise any input device arranged to receive input froma user. For example, keypad 256 may include a push button numeric dial,or a keyboard. Keypad 256 may also include command buttons that areassociated with selecting and sending images.

In one embodiment, a light source driver interface 258 may providesignal interface with the light source driver 102. Light source driverinterface 258 may be used to provide modulation and timing controlinformation to light source driver 102. For example, if CPU 222 is usedas processor 116 for processing feedback and timing information of IPD100, then light source driver interface 258 may be used to deliver imagecontrol 124 and timing information 126 to light source driver 102.

Video interface 259 may generally be used for providing signals of aparticular type and/or formatting images for display on a particulartype of display device 254. For example, if display 254 is a raster typedevice, such as a CRT, then video interface 259 provides the appropriatesignal timing, voltage levels, H-sync, V-sync, and the like to enablethe image to be displayed on display 254. If display 254 is IPD, thevideo interface 259 may implement some or all of the components shown inFIG. 1 to enable an image to be displayed using IPD 100.

Computing device 200 may also include input/output interface 260 forcommunicating with external devices, such as a headset, or other inputor output devices not shown in FIG. 2. Input/output interface 260 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, or the like.

In one embodiment, detector interface 262 may be used to collect timingand screen position information from detector 112 and passing suchinformation on to processor 116 and/or CPU 222 for further processing.In another embodiment, detector interface 262 may be integrated withvideo interface to 259 as one unit. In yet another embodiment, detectorinterface 262 may be external to computing device 200 and be part of anintegrated external IPD unit.

GPS transceiver 264 can determine the physical coordinates of computingdevice 200 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 264 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS or thelike, to further determine the physical location of computing device 200on the surface of the Earth. It is understood that under differentconditions, GPS transceiver 264 can determine a physical location withinmillimeters for computing device 200; and in other cases, the determinedphysical location may be less precise, such as within a meter orsignificantly greater distances. In one embodiment, however, mobiledevice may through other components, provide other information that maybe employed to determine a physical location of the device, includingfor example, a MAC address, IP address, or the like.

Mass memory 230 includes a RAM 232, a ROM 234, and other storage means.Mass memory 230 illustrates another example of computer storage mediafor storage of information such as computer readable instructions, datastructures, program modules or other data. Mass memory 230 stores abasic input/output system (“BIOS”) 240 for controlling low-leveloperation of computing device 200. The mass memory also stores anoperating system 241 for controlling the operation of computing device200. It will be appreciated that this component may include a generalpurpose operating system such as a version of UNIX, or LINUX™, or aspecialized computing communication operating system such as WindowsMobile™, or the Symbian® operating system. The operating system mayinclude, or interface with a Java virtual machine module that enablescontrol of hardware components and/or operating system operations viaJava application programs. In one embodiment, mass memory 232 may beused as memory 118, for holding an image to be displayed using IPD 100,coupled with processor 116 and/or CPU 222.

Memory 230 may further include one or more data storage 244, which canbe utilized by computing device 200 to store, among other things,applications 242 and/or other data. For example, data storage 244 mayalso be employed to store information that describes variouscapabilities of computing device 200, a device identifier, and the like.The information may then be provided to another device based on any of avariety of events, including being sent as part of a header during acommunication, sent upon request, or the like.

Applications 242 may include computer executable instructions which,when executed by computing device 200, transmit, receive, and/orotherwise process messages (e.g., SMS, MMS, IMS, IM, email, and/or othermessages), audio, video, and enable telecommunication with another userof another computing device. Other examples of application programsinclude calendars, browsers, email clients, IM applications, VOIPapplications, contact managers, task managers, database programs, wordprocessing programs, security applications, spreadsheet programs, games,search programs, and so forth. Applications 242 may further includeimage processor 243, beam trajectory processor 245, and detectorprocessor 247.

In one embodiment, image processor 243 is a software component that mayperform functions associated with processing digital images for displayusing IPD 100. For example, image processor 243 may used to fetch animage to display using IPD 100 from memory 118. In one embodiment, imageprocessor 243 works in collaboration with other components, such asvideo interface 259, detector interface 262, beam trajectory processor245, and detector processor 247. Additionally, image processor 243 maybe used to perform digital image processing operations on the imagefetched from memory 118. For example, image processor 243 may use anadjustment coefficients matrix to adjust each color component of eachimage pixel of the image in memory 118 before using IPD 100 to displaythe image. In one embodiment multiple adjustment coefficients matricesmay be used to adjust different image parameters. For example, onematrix may be used to adjust brightness, while another matrix may beused to adjust saturation when using HSB color space. In one embodiment,RGB color representation may be converted to HSB, be adjusted accordingto the appropriate coefficient matrices, and converted back to RGB formodulation.

Those skilled in the art will appreciate that there are many imageprocessing operations that may be performed on an image beforedisplaying the image. For example, various filtering operations may beperformed on the image to filter out noise, sharpen edges, and improvecontrast. Such image processing operations may be based on the timing oftracer beam 122 pulses, the detected view angle of a viewer, the textureof screen 114, and the like.

Beam Trajectory Processor (BTP) 245 is used to process information aboutthe pseudorandom scanline trajectory based on tracer beam pulses 122.BTP 245 may work in cooperation with image processor 243 to estimate thenext screen position for projecting the image. In one embodiment, BTP245 collect data about N successive tracer beam 122 pulses from acurrent scanline being scanned by scanner 110. Next, BTP 245 uses the Nsuccessive pulses to estimate the next M display screen positions on thecurrent scanline. For example, BTP 245 may use numerical methods to fita curve through the [X, Y] positions of the N successive pulses andestimate/predict the next M display screen positions on the currentscanline. Those skilled in the relevant arts will appreciate that thereare many numerical methods that may be used for this application. Forexample, least squares curve fit may be used to minimize curve fiterrors. Subsequently, image processor 243 fetches image pixels frommemory at 118, corresponding to the M display positions on the currentscanline.

In one embodiment, the screen position estimation/prediction processdescribed above may be repeated for each tracer beam 122 pulse that isdetected by detector 112. In effect, a sliding window type algorithm maybe used to continuously and accurately estimate and update the predictedposition of the next screen position before the scanline actuallycrosses the predicted screen position. In this embodiment, the width ofthe sliding window is the N screen positions.

In one embodiment, detector processor 247 may work in collaboration withdetector interface 262 to collect and preprocess data for tracer beam122 pulses before providing such data to BTP 245 for trajectoryestimations. Those skilled in the relevant arts will appreciate thatimage processor 243, BTP 245, and detector processor 247 may beintegrated into one component. Similarly, the functions performed byeach of these components may be decomposed and distributed over othercomponents, software or hardware, of computing device 200.

Those skilled in the relevant arts will appreciate that some of thecomponents described above with respect to FIG. 2, for example, beamtrajectory processor 245, image processor 243, and detector processor247, may be integrated together in a single more comprehensive componentthat performs the functions of the individual components described.Conversely, some of the components may be decomposed and distributedover multiple smaller components with more focused functions.Additionally, some of the components described above may be implementedin hardware, software, firmware, or a combination of these.

Generalized System Operation

FIG. 3 shows one embodiment of IPD 100 control subsystem. The controlsubsystem takes feedback information from detector 112, processes thefeedback information in conjunction with image data stored in memory118, and provides control information to light source driver 102. In oneembodiment, the control subsystem includes a timing control 304outputting a tracer beam timing and/or frequency control signal, t_(s),and another reference timing signal, t_(ref), to an intensity controlcomponent 302. The intensity control component 302 provides modulationcontrol signals 308 for modulating component image beams generated bylight sources 104.

In one embodiment, timing control component 304 is implemented as partof BTP 245 and/or a detector processor 247. In another embodiment,timing control component 304 is implemented as a separate hardwarecircuit that may interface with detector interface 262. Similarly, inone embodiment, intensity control component 302 may be implemented aspart of image processor 243. In another embodiment, intensity controlcomponent 302 may be implemented as an independent component that iscoupled with image processor 243.

In one embodiment, timing control component 304 takes as input timinginformation to from to detector 112 and outputs tracer beam 122 timinginformation, such as t_(s) with pulse period T between pulses 306.Tracer beam 122 timing information may subsequently be used forgenerating tracer beam 122 via tracer beam generator 106. Timing controlcomponent 304 may also calculate pulse flight time by subtracting isfrom to. Pulse flight information may be used to calculate the distanceof screen 114 from scanner 110, based on which image intensity may becontrolled. Timing control 304 may also be used to vary pulse period Tand thus, vary the effective resolution of the image projected using IPD100 dynamically. Dynamic, localized, real-time variation of imageresolution may be useful to adjust the displayed image on an unevensurface of screen 114. For example, if the surface of screen 114 has anedge with a sharp drop or angle, such as a wall corner with a largedrop, at a given resolution, a display pixel may fall partly on the topside of the edge and partly on the bottom side of the edge, thus,splitting and distorting the displayed pixel. However, if the displayedpixel is split into two pixels by increasing local resolutiondynamically, then instead of splitting the pixel over the edge, onepixel at appropriate intensity is projected on the top side of the edgeand another pixel and at another appropriate intensity is projected onthe bottom side of the edge.

In one embodiment, dynamic resolution adjustment in conjunction withother features of IPD, such as feedback and flight time information, maybe used to project an image on several odd-shaped or angled wallssurrounding the IPD for creating a visually immersive environment. Suchvisually immersive environment may be used in video games where theplayer is at the center of the room in a virtual game environment. Inone embodiment, a single IPD may be used to project an image about 180°wide. In another embodiment, more than one IPD may be used to project animage about 360° wide (for example, planetarium style projection),completely surrounding a viewer.

Another application of dynamic resolution control is focusing highresolution, and thus, high quality, where high resolution is neededmost. For example, an image having details of a face against a blue skybackground can benefit from the high resolution for showing facialwrinkles, hair, shadows, and the like, while the blue sky background canbe displayed at a relatively lower resolution without sacrificingquality significantly or noticeably.

In one embodiment, intensity control component 302 determines intensityof component image beams based on pixel values of the image in memory118 corresponding to the next M screen positions on the currentscanline. The intensity of each component image beam for each predictedscreen position may be further adjusted by adjustment coefficients in animage processing matrix. In one embodiment, detector 112 includes acolor camera that detects combined image beam 120 from screen 114, inaddition to detecting tracer beam 122, as another feedback signal.Detected combined image beam 120 may be used to further adjust displayedimage quality by comparing detected combined image beam 120 at eachscreen position with the corresponding image pixel in memory 118.

In one embodiment, the adjustment of the displayed image at each screenposition may be done over successive scan cycles, each scan cyclesweeping a new pseudorandom scanline, of the same pixel. As scanner 110randomly projects scanlines onto screen 114, eventually each screenposition is scanned again in a later scan cycle, providing theopportunity to further adjust the color and intensity of the projectedimage at each screen position, or add additional detail, possiblyfilling in small spaces missed in the previous scan cycles, to sharpenor soften edges as needed. Due to the high scan rate of scanner 110,each display pixel is likely scanned and adjusted multiple times at arate that is imperceptible to a human viewer. The human visual systemintegrates successive adjustments of color and intensity into aperceived whole by averaging the color and intensity over time if suchadjustments occur fast enough for the human visual system. Thus, even ifin one scan cycle, the color and intensity of a display pixel is lessthan perfect as compared with the corresponding pixel of the image inmemory 118, for example, due to lighting and/or screen 114 surfaceimperfections, the color and intensity of the display pixel is adjustedin the next few scan cycles and integrated by human eye before the humanvision has a chance to perceive the imperfection in one cycle.

FIG. 4A shows one embodiment of pseudo random beam trajectoriesgenerated by IPD 100. IPD type projector 402 is used to projectpseudorandom scanlines with trajectories 404 onto screen 114. Asscanline 404 trajectories randomly cover surface of screen 114 hundredsof thousands of times a second, each screen position is scanned hundredsof times per second. Each display pixel may fall on different scanlinesduring scanning because of the pseudorandom nature of the scanlines. Forexample, a pixel 408 may fall on one scanline during one scan cycle andfall on another scanline passing through the same point in another scancycle. This is so because it is well known in basic geometry thatinfinitely many lines can pass through the same point in a plane, suchas the surface of screen 114.

As an illustrative example, consider an image 406 of a face that isdisplayed using IPD 402 on screen 114. Each screen position 408 on theface 406 may fall on one or more scanlines. When screen position 408 ispredicted to be the next display position on a current scanline,processor 116 (or equivalently, image processor 243 shown in FIG. 2)fetches the corresponding pixel data from the image in memory 118. Thecorresponding pixel data from the image in memory 118 are then used tomodulate component image beams output by light sources 104 that aresubsequently combined to form combined image beam 120. Combined imagebeam 120 is reflected by scanner 110 onto screen 114 at screen position408 when the current scanline reaches screen position 408. Thisoperation is typically performed on a nano-second time scale.

For relatively large spans of uniform image portions on a pixel scale,for example, a portion of blue sky or white wall, the modulation ofcomponent image beams need not change because the same color andintensities are repeated for many contiguous pixels. ARun-Length-Limited (RLL) image coding and/or display algorithm may beused to reduce the amount of processing needed to display such images.Most graphical images, such as various pictures like sceneries,clothing, faces, and the like, have large spans of uniform image colorsand intensities at pixel level and can benefit from RLL based processingto increase efficiency. Even larger spans of uniformity may beencountered in synthetic game scenes and software application graphicssuch as desktop business applications, web pages, and the like.

FIG. 4B shows an embodiment of one pseudo random beam trajectory withscanned and predicted trajectory portions. The process of projecting animage using the IPD may be divided into two distinct phases: a feedbackphase during which an already-scanned portion of scanline 404 isdetected, and a projection phase during which combined image beam 120 isprojected onto a predicted portion of scanline 404. Correspondingly,scanline 404 has two distinct portions, one, scanned beam trajectoryportion 416, and two, predicted trajectory portion 410. Scanned beamtrajectory portion 416 of scanline 404 ends at current beam position 412and includes a sequence of pulses 414, typically generated on the basisof nano-second timing. The predicted trajectory portion 410 is theportion that is predicted by processor 116 based on the data associatedwith the sequence of pulses 414. The data associated with the sequenceof pulses 414 include [X,Y] position of the display pixel on whichpulses 414 are projected, the time at which pulses 414 are detected, andthe like. In effect, pulses 414 define multiple points which specify ascanned portion 416 of scanline 404. The remaining predicted trajectoryportion 410 of scanline 404 is predicted or estimated by numericaltechniques, such as curve fits, based on the data associated with pulses414. Combined image beam 120 is generally projected on predictedtrajectory portion 410 of scanline 404.

FIG. 5 shows an embodiment of the IPD of FIG. 1 depicting one image beamand tracer beam point. This embodiment includes a projector 402including a light source and optics unit 508, a detector 112 coupledwith a lens 510, ad a feedback control unit 502 outputting controlsignals 506. A combined image beam 512 is projected onto screen 114 atscreen position 408. screen position 408 is thus defined by two lightbeams in close proximity: a projection X of combined image beam 120 anda projection O of tracer beam 514. In one embodiment, the tracer beam514 includes a train of IR pulses 418 projected on screen 114 as thecurrent scanline is swept across screen 114. In one embodiment,projection O of IR pulse 418 is co-centric with projection X of combinedimage beam 512. In another embodiment, projection O of IR pulse 418 ispositioned side-by-side with respect to projection X of combined imagebeam 512. In yet another embodiment, tracer beam 514 is ashort-duration, on the order of a few nano-seconds, visible light pulse,that is imperceptible to human vision because of its very shortduration.

A reflection 516 of tracer beam 514 is detected by detector 112 which isgenerally positioned in close proximity to light sources and optics 508,for example, next to scanner 110 (not shown in FIG. 5). Lens 510 may beused to collect and focus an image of reflected light 516 from screen114 onto the sensor array, for example, a CCD array, of detector 112. Inone embodiment reflected light 516 also includes a reflection ofcombined image beam 512 for comparison of the displayed image with theimage in memory 118 (see FIG. 1). Detector 112 provides feedbackinformation, such as timing information to of a pulse 414 (see FIG. 4B),position information [X, Y], and image information, such as displaypixel color and intensity data, to feedback control component 502. Inone embodiment, with reference to FIGS. 1 and 2, feedback controlcomponent 502 is implemented as one or more of processor 116, CPU 222,detector interface 262, image processor 243, BTP 245, and detectorprocessor 247.

Feedback control 502 provides control information 506 to light sourcesand optics component 508. Such control information include timinginformation t_(s), and modulation control information 308 (see FIG. 3)to control the generation of tracer beam 514 and combined image beam512, as more fully described above.

In one embodiment, with reference to FIGS. 1 and 2, light sources andoptics component 508 may be implemented as one or more of light sourcedriver 102, light sources 104, tracer beam generator 106, beam combiner108, and scanner 110.

FIG. 6A shows an embodiment of an application of the IPD of FIG. 1 withdifferent user vantage points. For example, a user 604 may be atposition-1, with respect to IPD 402, and later move to position-2. Inone embodiment, a head-mounted or body mounted position sensor 602 maybe used to provide feedback about the user 604's current position to IPD402. In one embodiment, position sensor 602 may be a position camerathat can collect visual information, such as an area of focus of user604 or other information about the image from the screen. Based onfeedback that IPD 402 obtains from detector 112 (see FIG. 1) and thefeedback provided by position sensor 602, processor 116 can calculateand adjust projected imaged beam 606 onto screen 114 such that reflectedbeam 608 observed by user 604 provides a proper perspective of theprojected image to user 604. For instance, if IPD 402 is projecting animage of a car viewed from front when user 604 is viewing screen 114from a vantage point along the direction of projection, then user 604sees a front view of the car. Now, if user 604 moves to position-1 tothe right of IPD 402, then IPD 402 calculates the new viewing angle forposition-1 and projects the right-side perspective of the car image foruser 604. Similarly, if user 604 moves to position-2 to the left of IPD402, then IPD 402 calculates the new viewing angle for position-2 andprojects the left-side perspective of the car image for user 604.

As described for FIG. 6A, this capability is useful in immersiveapplications, for example, video games, where user perspective can bedynamically and in real-time updated and projected. Additionally, in oneembodiment, each eye can be treated as a separate point of view withdifferent left and right depth perceptions. In this case, eyewear withseparate shutters for each eye could be used to control the image viewedby each eye. Alternatively, polarization of lens in front of each eyecould be employed to control the image viewed by each eye.

FIG. 6B shows an embodiment of a IPD response to different viewingperspectives. In this embodiment, an illustrative image of a car isprojected by the IPD onto a screen. When user 604 is at position-1, witha line of sight 624, which is substantially perpendicular to the screen,that is, substantially parallel with a center-line of projection fromthe IPD to the screen, a front-view image 620 of the car isautomatically projected by the IPD. The position of user 604, in thiscase, position-1, relative to the center-line of projection isdetermined by information communicated via position sensor 602 (see FIG.6A). When user 604 changes his position to position-2, with a line ofsight 626, which is at an angle with respect to the center-line ofprojection, that is, not parallel with the center-line, the IPD detectsthe new position via information provided by position sensor 602.Subsequently, the IPD adjusts the image of the car to project thecorrect perspective image 622 of the car on the screen, as if user 604is looking at a physical car in real world in 3-D. This embodimentenables projection of images in an immersive environment, such as videogames, virtual reality applications like moving in and around an object,car, or building, and the like.

In one embodiment, the feedback and image adjustment mechanisms of IPDmay be used in an interactive virtual and/or augmented reality (VAR)system for interaction with user 604. In this embodiment, when user 604approaches a projected object on the screen, the IPD may provide manyVAR capabilities, such as “zooming in,” revealing ever greater detailabout the projected object from the correct perspective, “zooming out,”“panning left/right,” “panning up/down,” handling a projected object,for example, lifting or pushing, and most other interactions that may beperformed on a real object. Furthermore, since few or no physicalconstraints exist in a VAR system, some interactions that cannot beperformed with a physical object may be performed on a projected object.For example, user 604 may walk through a wall without breaking the wall,as light passing through glass. A “zoom factor” may determine the rateof magnification as a function of the distance between user 604 and theprojected object. User 604 may approach any object in view on thescreen. The zoom factor may depend on the relative distance and desiredrealism in the VAR system. These features may create enhanced depth &motion experience.

In one embodiment, the IPD renders a scene by continuous real-timeadjustments of the projected image by referencing the image as viewedthrough a position camera aligned with the viewer's perspective, asdescribed above with respect to FIGS. 6A and 6B. In one embodiment, theposition camera may be focused on a field of view within the screen,representing a focus area of the viewer/user. For example, if the screenhas the image of a football match, and the position camera is focused ona subarea of the screen, where the ball is located, then that subarea isthe field of view. In one embodiment, the position camera may be mountedon goggles such that the direction of the gaze of the viewer/userdetermines the field of view seen by the position camera, which issubsequently fed back to the IPD.

The position feedback and independence of quantization allows the IPD torender objects in great detail, relatively unconstrained by any fixedresolution. By concentrating the graphics subsystem computationalresources, such as memory and processing power, on the rendering ofgraphics polygons describing the projected object, more detailed objectsmay be rendered. The system resources are more efficiently used becausethe field of view may be rendered with greater detail than a peripheralfield, while still a sufficiently realistic peripheral vision experienceis maintained. The peripheral field may be rendered in less detail, orwith greater latency (for example, due to a lower processing priority),conserving computational resources such as memory and computingbandwidth.

As discussed above, the distance of the viewer/user to the projectedsurface may be determined by detecting the tracer beam and determiningthe flight time of the tracer beam pulses. A total viewer distance,D_(T) to an object is the sum of a real distance, D_(R), and animaginary distance, D_(I):D_(T)=D_(R)+D_(I). D_(R) is the distance fromthe viewer (for example, the viewer's eyes) to the position on thescreen where the object is projected. D_(I) is the distance of theobject away from and behind the plane represented by the projectionsurface (for example, screen), measured along the direction of view. Forexample, in a video game or VAR environment, a game engine may controlD_(I) by the varying size, shading, and rendering of perceivable detailsappropriately. D_(T), may also be referred to as a radial distance(total perceived distance), that is the distance from the object in viewto the viewer's eye measured radially, that is, along the direction ofview. In addition to this radial distance, a correct angular perspectivemay also be determined, as discussed above.

Objects that can be rendered and viewed in different directions andangular positions may be limited by the screen geometry and the positionof the viewer. In case of a standard single rectangular projectionsurface (for example, normal front or rear projection systemconfigurations) the limiting factors are screen height and width, andthe viewing distance. If projection is done on walls surrounding theviewer, little or no geometric limitations may exist for the field ofview of the user and interactions with objects projected onto thescreen. Apart from screen limitations imposed by geometry or physicalconfiguration, the viewer, equipped with the position camera, may movefreely about the screen and approach any object in his field of view,which is usually a limited subset of, a “view cone,” within the totalpossible field of view. When the viewer approaches the screen theobjects in his view will tend to naturally increase in size because theviewer is getting closer to them. As the image on the screen getsbigger, the image also occupies a grater portion of the field of view ofthe viewer. Accordingly, the viewer expects to see more details aboutthe object, just as in real world.

At this point certain problems may occur that are best illustrated withan example. In an illustrative example, the viewer is looking at astatic outdoor scene including a mountain about 10 miles away, sometrees, and some rocks. Without any image adjustments by IPD, as theviewer moves closer to the screen, the distance of the viewer to anobject, for example, a tree, is reduced exactly proportionally to thereal distance traversed by the viewer towards the screen. Thus as theviewer approaches a far away object, for example, the mountain 10 mileaway in the view, the mountain will appear to come much closer than itshould. For example, if the viewer travels three feet, half the distanceof six feet from his original position to the screen, he will see themountain twice as close even though he has not actually traveled fivemiles (half the perceived distance to the mountain). Even though thischange in perceived distance is an expected artifact in a static image,it immediately tells the viewer's brain that the mountain is actuallyjust a number of pixels projected on a flat surface six feet away, andnot a real mountain 10 miles away. Thus, as the viewer comes closer tothe screen, his natural depth perception is violated by every object inthe scene, except possibly those objects that are perceived as close tothe screen in the foreground.

Additionally, without image adjustment, all objects in the projectedview may have unaltered geometric positions fixed to the projectionsurface, which is clearly not what would happen in reality. Furthermore,in the far background, the horizon does not recede, as it does in thereal world, but comes 50% closer as the viewer moves three feet. In realworld, as the user walks towards a mountain in the distance andapproaches closer objects, the angular positions of objects with respectto the viewer's field of view generally increase for all objects in theview, more for closer objects, and less for farther objects.

Furthermore, without image adjustment, details that were hard to see atsome distance, for example a tree that was 15 feet away, are still hardto see, when the viewer gets three feet closer. So, getting closer hasless effect than expected. Also, as the viewer gets closer to thescreen, serious projection and display limitations become visible due toan excessively coarse image rendering granularity and pixel pitch andother quantization artifacts that destroy the “suspension of disbelief”.

The above visual faults may be more acute in a VAR environment. Thevisual faults that result from lack of proper adjustment to theprojected image when the viewer moves with respect to the screen resultin the loss of the carefully constructed illusion, which so critical toa full immersive experience.

The IPD provides the capability to properly adjust the projected images,static, dynamic, or interactive, like VAR and video game environments,with respect to the viewer's position and field of view, to provide arealistic visual experience. Continuing with the illustrative exampleabove, the viewer standing six feet away from the screen, may see in thecenter of the field of view a far away object, for example, themountain. As the viewer moves half the distance to the screen—threefeet—the system detects the viewer's motion and adjusts the screen sizeof the mountain correctly to half size, canceling the effect of theviewer moving 50% closer to the screen. The mountain is seen by theviewer as unaltered, as he would also see in real world, since themountain is still approximately 10 miles away (less three feet) but theimage is viewed from half the original distance to the screen. Withoutdetecting the viewer's position, the mountain's apparent size would havedoubled, as discussed above.

At the same time, a small bush in the foreground is left unaltered bythe IPD since the viewer's motion in fact results in the real distanceto that bush being halved and the unaltered projected image should looktwice as big, at half the distance, precisely as it would if it were areal bush. Additionally, since the bush is now quite close, a systemusing the IPD, for example the VAR system, allocates some extra graphicsresources to render more visible details in each leaf of the bush, andthe like, to render a closer view of the bush, since the user is nowcloser and expects to see more detail. This is made possible by thedynamic resolution control feature of the IPD. Possibly, as the viewergets very close, the system may actually project a shadow of the viewerover the bush. Such rendering would further enhance the realismexperienced by the viewer. The realism is further supported by the IPDdue to substantial lack of fixed pixel size, pixel position, pixelorientation, or scan patterns. The IPD may render and display allobjects in view in the natural looking detail with few or no obviousvisual artifacts. An important point to note is that the IPD isindependent of a game controller, buttons, or keys to adjust the image.The IPD may adjust the image automatically based on the position cameraand the field of view of the viewer.

FIG. 6C shows an embodiment of a IPD projection onto a tilted screen.IPD 402 may project an image onto screen 114 when centerline 650 ofprojection is substantially perpendicular to the surface of screen 114.In this configuration, symmetrical projection lines 642 and 644 aresubstantially equal in length due to the symmetry of projection withrespect to screen 114. In one embodiment, IPD 402 may project the imageonto screen 640 having an angle not perpendicular to centerline 650. Inthis configuration, asymmetrical projection lines 646 and 648 aresubstantially different in length due to the asymmetrical configurationof screen 640 with respect to IPD 402. Because asymmetrical projectionlines 646 and 648 have different lengths, the flight time of tracer beampulses 418 (see FIG. 4B) will be different when the tracer beam isprojected along the asymmetrical projection line 646 than when thetracer beam is projected along the asymmetrical projection line 648.This difference in flight time may be detected by the IPD and used toadjust the projected image and avoid image distortion due to a“stretching” effect of a tilted screen.

The stretching effect is caused by geometric projection of a linesegment onto a plane having an angle α with respect to the line segment.The length of the projected line segment, L_(p)=L/Cos α, where L is thelength of the line segment. In this case, L_(p)>L, causing thestretching of the line segment. Similarly, any other shape projected ona tilted screen is also stretched by the same factor of 1/Cos α. Thesame adjustments done for tilted screen 640 may be applied to aprojection surface with small surface irregularities and/or angles, suchas a fabric with wrinkles or angled walls, in a piece-wise fashion. Thisability of IPD to use the tracer beam feedback and automatically adjustthe projected image in real-time enables projection of images ontouneven and irregular surfaces with substantially reduced or eliminateddistortions.

FIG. 7A shows an embodiment of a mobile device with an embedded IPD.Mobile device 702 may be any of a variety of mobile computing devices,such as mobile phones, PDA's, laptop PC's, and the like. The operationof IPD is similar to that described above. Because IPD technology issmall by nature, using miniaturized solid state components such as LEDlight sources 104, MEMS (Micro Electronic and Mechanical Systems), suchas scanner 110 (see FIG. 1), processor 116, and the like, and the actualscreen, such as screen 114, where the image is projected is generallyexternal to IPD 100, IPD is suitable for housing in small physicalcomputing devices with little or no loss of screen size and/or displayedimage quality. One limitation of small mobile computing devices isavailability of sufficient power to run light sources for high intensityprojection. In one embodiment, an AC power adapter may be used toprovide additional electrical power for brighter or longer durationprojections using mobile device 702. Software applications that needhigh quality GUI on modern mobile devices and can benefit from the IPDtechnology include electronic mail (e-mail), video games, mobile webpages, and the like. The outputs of such software applications may bedisplayed using a IPD as a projected image 704, instead of using thegenerally small and low resolution screens available on such devices.

FIG. 7B shows another embodiment of a mobile device with an embedded IPDand a head-mounted position sensor. In one embodiment, position sensor602, shown in FIG. 6, may be implemented as an ear-mounted device thatcommunicates wirelessly with the IPD embedded in mobile computing device702 to provide wireless position feedback 708 to the IPD, as describedabove with respect to FIG. 6. Displayed image perspective is adjusted asuser 604 moves around with respect to screen 114 and/or mobile device702 when set in a stationary position.

FIG. 8A shows a flow diagram of one embodiment of a high level processof generating an image using a IPD. The process starts at block 880 andproceeds to block 882 where a tracer beam is projected onto a projectionscreen along a pseudorandom scanline trajectory. As described above, thetracer beam may include intense, short-duration light pulses, such as IRpulses, that may be imperceptible to human vision but may be easilydetected by a detector. The process moves to block 886.

At block 886, the tracer beam is detected. Various detection schemes maybe used that detect the screen position of each pulse on thepseudorandom scanline, such as 2-D CCD arrays, beam-folding opticaldetectors, and the like. Several screen positions corresponding to thetracer beam pulses are detected. In one embodiment, a sliding windowwith a width of N screen positions may be used to detect the tracerbeam. The N screen positions may subsequently be used to predict thetrajectory of the scanline. The process proceeds to block 888.

At block 888, a portion of the scanline trajectory that is not yetprojected by the IPD is predicted based on the N screen positionsdetected at block 886. For example, a curve fit algorithm may be used toextrapolate and determine the next M screen positions on the scanline.The process proceeds onto block 890.

At block 890, stored image pixels or generated graphics corresponding tothe next M screen positions are obtained from an image source, such asfrom memory coupled with the IPD. The correspondence of image pixelswith the next M screen positions are not necessarily one-to-one. Forexample, one image pixel may cover and correspond to multiple screenpositions, depending on the resolution of the image to be displayed. Theobtained image is then projected by the IPD onto one or more of the Mscreen positions. In one embodiment, the next M screen positions aredetermined on a continuous basis, for example, using another slidingwindow with a width of M. The process terminates at block 892.Additional process details are described below with respect to FIG. 8B.

FIG. 8B shows a flow diagram of one embodiment of a process ofgenerating an image with the IPD of FIG. 1. With reference to FIGS. 1and 8, the overall process of generating an image using a IPD starts atblock 800 and proceeds to block 805 where tracer beam 122 pulses areprojected onto screen 114. As discussed above, tracer beam 122 may be IRpulses that are projected in parallel with combined image beam 120 ontoscreen 114 and subsequently detected by detector 112. Tracer beam 122 isused to predict the next screen position on the current scan line beingswept across screen 114 by scanner 110. The process proceeds to block810.

At block 810, detector 112 is used to detect tracer beam 122 pulses, andprovide the raw data and/or preprocessed data associated with the pulsesto processor 116. In one embodiment, a sliding window algorithm isutilized to collected data about the N preceding pulses correspondingwith the N preceding screen positions. The process proceeds to block815.

At block 815, processor 116 calculates the trajectory of the currentscan line to predict and/or estimate the next screen position on thecurrent scanline for display of image pixel from memory 118. In oneembodiment, next M screen positions are predicted based on the Npreceding tracer beam 122 pulses on the current scanline. In oneembodiment, the determination/prediction of the next M screen positionsis repeated for each of the M pixels as the current scanline sweepcontinues by the scanner 110, thus implementing a sliding windowalgorithm both at the feedback end where data are collected about Npreceding pulses and at the prediction end where M next screen positionsare predicted. The sliding window at the feedback end is N pulses widewhile the sliding window at the prediction end is M pixels wide. Theprocess moves on to decision block 820.

At decision block 820, the process determines whether adjustmentcoefficients are updated and whether pixel color values, such asintensity or saturation, need adjustment before display. If so, theprocess proceeds to block 825 where the pixel values obtained frommemory 118 are adjusted using the adjustment coefficients beforeproceeding to block 830. Otherwise, the process proceeds directly toblock 830.

At block 830, the next M screen positions on the current scanline aredetermined. As noted above, in one embodiment, the next M screenpositions are predicted based on dual sliding windows, one at thefeedback end where data about preceding N screen positions arecollected, and one at the prediction end where each of the next M screenpositions are determined based on the data collected about the precedingN screen positions. The process proceeds to block 835.

At block 835, component image beams outputted by light sources 104 aremodulated according to the corresponding values of color components ofthe pixels of image in memory 118, where the pixels correspond to thenext screen position on the current scanline predicted at block 830. Forexample, the intensity of the R (Red) component image beam may be set tothe red component value of the image in memory 118 for the pixel to bedisplayed next on the current scanline. The process proceeds to block840.

At block 840, the component image beams, for example, RGB components,are combined together to form one combined image beam 120. In oneembodiment, a prism may be used to combine the component image beam. Inother embodiments other methods currently known in the art or methods tobe discovered in the future may be used to combine the component imagebeams. The process proceeds to block 845.

At block 845, a scanner 110, for example a MEMS device with a rotatingmirror with two degrees of rotational freedom, for example, twoorthogonal planes of rotation, reflects the combined image beam 120 aspseudorandom scanlines sweeping across screen 114. Different methods maybe used to inject randomness into the direction of the scanlines fromone scan cycle to the next. These methods range from mechanical andphysical means, such as imprecisely controlled off-center vibrations atmicroscopic level, to electronic and software means, such as randomnumber generators. The process proceeds to block 850.

At block 850, detector 112 or another image detection camera optionallydetects the reflection of combined image beam 120 from screen 114. Inone embodiment, combined image beam 120 is scattered off screen 114 andis refocused onto detector 112, or the other image detection camera,using a lens 510 (see FIG. 5). Data collected about projected pixelvalues and positions are used to improve the projected pixel values (theimage) on the next scan cycle. The process proceeds to decision block855.

At decision block 855, the data collected about projected pixel valuesare compared with the corresponding pixel value of the image stored inmemory 118 to determine any deviations. If any deviations are detected,for example, because of screen 114 color or texture, adjustmentcoefficients for the pixel values of the image in memory 118 are updatedat block 860 to adjust such pixel values for the next scan cycle.Otherwise, the process proceeds to block 805 and the process is repeatedfor the next screen position on the current scanline. As one scanline iscompleted, another scanline is started and the same process describedabove is repeated with respect to the new scanline.

It will be understood that each block of the flowchart illustration, andcombinations of blocks in the flowchart illustration, can be implementedby computer program instructions. These program instructions may beprovided to a processor to produce a machine, such that theinstructions, which execute on the processor, create means forimplementing the actions specified in the flowchart block or blocks. Thecomputer program instructions may be executed by a processor to cause aseries of operational steps to be performed by the processor to producea computer implemented process such that the instructions, which executeon the processor to provide steps for implementing the actions specifiedin the flowchart block or blocks. The computer program instructions mayalso cause at least some of the operational steps shown in the blocks ofthe flowchart to be performed in parallel. Moreover, some of the stepsmay also be performed across more than one processor, such as mightarise in a multi-processor computer system. In addition, one or moreblocks or combinations of blocks in the flowchart illustration may alsobe performed concurrently with other blocks or combinations of blocks,or even in a different sequence than illustrated without departing fromthe scope or spirit of the invention.

Accordingly, blocks of the flowchart illustration support combinationsof means for performing the specified actions, combinations of steps forperforming the specified actions and program instruction means forperforming the specified actions. It will also be understood that eachblock of the flowchart illustration, and combinations of blocks in theflowchart illustration, can be implemented by special purposehardware-based systems which perform the specified actions or steps, orcombinations of special purpose hardware and computer instructions.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. Since many embodiments of the invention can be made withoutdeparting from the spirit and scope of the invention, the inventionresides in the claims hereinafter appended.

What is claimed as new and desired to be protected by Letters Patent ofthe United States is:
 1. A system for projecting one or more images ontoone or more surfaces, comprising: one or more light sources; one or moresensors; a memory that stores at least instructions; and one or moreprocessor devices that execute the instructions that perform actions,comprising: employing the one or more light sources to project the oneor more images onto the one or more surfaces; employing the one or moresensors to determine a vantage point of an area of focus for a userviewing one or more perspectives of the one or more images projectedonto the one or more surfaces; updating a viewing angle of the one ormore perspectives of the one or more images based on the determinedvantage point of the area of focus for the user and one or moreadjustments to corresponding intensities of the one or more lightsources; and employing the one or more updated intensity adjustments tothe light sources to project the one or more updated images onto the oneor more surfaces based on the determined vantage point of the area offocus for the user.